Determining and representing health of cognitive systems

ABSTRACT

Technical solutions are described for representing health of a cognitive system. An example computer-implemented method includes displaying an animated set of icons, each icon representing a question input to the cognitive system. Each icon has a respective movement pattern. The computer-implemented method also includes receiving a selection of a first icon from the animated set of icons. The computer-implemented method also includes in response, identifying a category of a first question corresponding to the first icon, determining a subset of the icons corresponding to question from the category, and displaying connectors between the subset of the icons.

BACKGROUND

The present application relates to computer systems, and morespecifically, to determining and representing health of a computersystem that is used as a cognitive system.

Typically, machines, such as computer systems, have delivered businessand societal benefits by automating, tabulation, and harnessingcomputational processing and programming to enhance enterprise andpersonal productivity. Cognitive systems are a next chapter in machines.Cognitive systems are machines that use natural language processing andmachine learning to enable people and machines to interact morenaturally to extend and magnify human expertise and cognition. Acognitive system includes computer systems and other types of machinesthat learn and interact with humans and other machines to provide expertassistance to scientists, engineers, lawyers, and other professionals ina fraction of the time it now takes. The cognitive system, thus, helpspeople extend their expertise across any domain of knowledge and makescomplex decisions involving extraordinary volumes of fast moving BigData.

For example, in healthcare, the cognitive system such as IBM™ WATSON™for Oncology helps oncologists treat cancer patients with individualizedevidence-based treatment options by analyzing patient data againstthousands of historical cases trained through more than 5,000 doctor andanalyst hours. The cognitive system can help doctors narrow down theoptions and pick the best treatments for the patients. The doctor stilldoes most of the thinking. The cognitive system, in this case,facilitates the doctor to make sense of the data and help make theprocess faster and more accurate. For city leaders, such cognitivesystems can help them prepare for major storms, to predict electricaloutages, to plan evacuations, and to prepare emergency managementequipment and personnel to respond in the areas that will need it most.

Considering that the cognitive system plays significant roles inimportant fields, it is important to ensure that the cognitive system isoperating without failures. Accordingly, determining and representingthe health of the cognitive system is a technical problem that needs tobe addressed. Further, because the cognitive systems include cognitivecapabilities, determining and representing the health of the cognitioncapabilities of the cognitive systems is a technical problem addressedherein.

Determining and representing the health of the cognitive system is achallenging technical problem to solve using current standards andconventions. Traditional data visualizations such as static charts andgraphs are inadequate, due to the amount of multivariate data that thecognitive system handles, and is to be communicated. Staticvisualizations of the data generated by the cognitive system may becomedense and difficult to comprehend. Further, aesthetically, such staticvisualizations do not convey the intelligence of the cognitive system.

SUMMARY

According to one or more embodiments, a computer implemented method forrepresenting health of a cognitive system includes displaying ananimated set of icons, each icon representing a question input to thecognitive system. Each icon has a respective movement pattern. Thecomputer implemented method also includes receiving a selection of afirst icon from the animated set of icons. The computer implementedmethod also includes in response, identifying a category of a firstquestion corresponding to the first icon, determining a subset of theicons corresponding to question from the category, and displayingconnectors between the subset of the icons.

According to one or more embodiments, a system for representing healthof a cognitive system, the system includes] a memory, a user-interface,and a processor coupled with the memory and the user-interface. Theprocessor displays, via the user-interface, an animated set of icons.Each icon represents a question input to the cognitive system, whereeach icon has a respective movement pattern. The processor alsoreceives, via the user-interface, a selection of a first icon from theset. The processor, in response, identifies a category of a firstquestion corresponding to the first icon; determines a subset of theicons corresponding to question from the category; and displays, via theuser-interface, connectors between the subset of the icons.

According to one or more embodiments, a computer program product forvisually depicting health of a cognitive system includes a computerreadable storage medium. The computer readable storage medium includescomputer executable instructions to display, via a user-interface, ananimated set of icons, each icon representing a question input to thecognitive system. Each icon has a respective movement pattern. Thecomputer readable storage medium also includes instructions to receive,via the user-interface, a selection of a first icon from the set. Thecomputer readable storage medium also includes instructions to inresponse, identify a category of a first question corresponding to thefirst icon, determine a subset of the icons corresponding to questionfrom the category, and display, via the user-interface, connectorsbetween the subset of the icons.

BRIEF DESCRIPTION OF THE DRAWINGS

The examples described throughout the present document may be betterunderstood with reference to the following drawings and description. Thecomponents in the figures are not necessarily to scale. Moreover, in thefigures, like-referenced numerals designate corresponding partsthroughout the different views.

FIG. 1 illustrates an example analysis system that determines andrepresents the health of a cognitive system in accordance with one ormore embodiments.

FIG. 2 illustrates example components of the analysis system inaccordance with one or more embodiments.

FIG. 3 illustrates an example visualization of the health of a cognitivesystem in accordance with one or more embodiments.

FIG. 4 illustrates a flowchart of an example method for generating avisualization representing the health of a cognitive system inaccordance with one or more embodiments.

FIG. 5 illustrates an example visualization of the health of a cognitivesystem in accordance with one or more embodiments.

FIG. 6 illustrates an example visualization of the health of a cognitivesystem in accordance with one or more embodiments.

DETAILED DESCRIPTION

Disclosed here are technical solutions for determining and representinghealth of a cognitive system. The technical solutions herein provide adynamic, data visualization that facilitates a user to quickly gain animpression of the health of the cognitive system, particularly thecognitive capabilities of the cognitive system. As such the technicalsolutions are rooted in and/or tied to computer technology in order toovercome a problem specifically arising in the realm of computers,specifically cognitive systems.

In exemplary embodiments, the technical solutions determine andfacilitate visualizing the health of the cognitive system using a movingorganic form, similar to a flock of birds, animals, insects, or thelike. The technical solutions herein combine a backend data service witha front end user-interface to present a moving, three dimensional,visual that gives the user a sense of the health of the system, and ispleasing to watch.

FIG. 1 illustrates an example scenario 100 in which a user 105, such asa doctor or any other professional, uses a cognitive system 110, forexample to analyze medical data. An administrator 125 is responsible formanaging the health of the cognitive system 110. The administrator 125uses an analysis system 120 to determine and visualize the health of thecognitive system 110. The analysis system 120 includes and/or implementsthe technical solutions described herein.

In an example, the analysis system 120 analyzes the health of thecognitive system 110 and displays a visual representation of the healthfor facilitating the administrator 125 to identify an action that s/hemay take regarding the health of the cognitive system 110. The analysissystem 120 may include a computer system, such as a desktop computer, alaptop computer, a tablet computer, a computer server, a smartphone, orany other hardware. The analysis system 120 is coupled with thecognitive system 110 to facilitate the analysis system 120 to accessperformance data of the cognitive system 110. The analysis system 120accesses the performance data either wirelessly or in a wired mannerusing a communication network.

The performance data may include a query-log of the cognitive system110. For example, the cognitive system 110 receives a query from theuser 105 using natural language and responds to the query based on thecognitive capabilities. The query may be in natural language, in theform of text input, speech input, or any other kind of input. Thecognitive system 110 identifies a confidence value with which thecognitive system 110 generates a response (or answer) for the query. Forexample, the cognitive system 110 parses the query syntactically andsemantically. After parsing the query, the cognitive system 110determines the confidence value for the question. The cognitive system110 may store the confidence value in the query-log associated with anentry corresponding to the question.

FIG. 2 illustrates example components of the analysis system 120. Theanalysis system 120 includes hardware, such as electronic circuitry. Forexample, the analysis system 120 includes, among other components, aprocessor 205, memory 210 coupled to a memory controller 215, and one ormore input devices 245 and/or output devices 240, such as peripheral orcontrol devices, that are communicatively coupled via a local I/Ocontroller 235. These devices 240 and 245 may include, for example,battery sensors, position sensors (altimeter, accelerometer, GPS),indicator/identification lights and the like. Input devices such as aconventional keyboard 250 and mouse 255 may be coupled to the I/Ocontroller 235. The I/O controller 235 may be, for example, one or morebuses or other wired or wireless connections, as are known in the art.The I/O controller 235 may have additional elements, which are omittedfor simplicity, such as controllers, buffers (caches), drivers,repeaters, and receivers, to enable communications.

The I/O devices 240, 245 may further include devices that communicateboth inputs and outputs, for instance disk and tape storage, a networkinterface card (NIC) or modulator/demodulator (for accessing otherfiles, devices, systems, or a network), a radio frequency (RF) or othertransceiver, a telephonic interface, a bridge, a router, and the like.

The processor 205 is a hardware device for executing hardwareinstructions or software, particularly those stored in memory 210. Theprocessor 205 may be a custom made or commercially available processor,a central processing unit (CPU), an auxiliary processor among severalprocessors associated with the analysis system 120, a semiconductorbased microprocessor (in the form of a microchip or chip set), amacroprocessor, or other device for executing instructions. Theprocessor 205 includes a cache 270, which may include, but is notlimited to, an instruction cache to speed up executable instructionfetch, a data cache to speed up data fetch and store, and a translationlookaside buffer (TLB) used to speed up virtual-to-physical addresstranslation for both executable instructions and data. The cache 270 maybe organized as a hierarchy of more cache levels (L1, L2, and so on.).

The memory 210 may include one or combinations of volatile memoryelements (for example, random access memory, RAM, such as DRAM, SRAM,SDRAM) and nonvolatile memory elements (for example, ROM, erasableprogrammable read only memory (EPROM), electronically erasableprogrammable read only memory (EEPROM), programmable read only memory(PROM), tape, compact disc read only memory (CD-ROM), disk, diskette,cartridge, cassette or the like). Moreover, the memory 210 mayincorporate electronic, magnetic, optical, or other types of storagemedia. Note that the memory 210 may have a distributed architecture,where various components are situated remote from one another but may beaccessed by the processor 205.

The instructions in memory 210 may include one or more separateprograms, each of which comprises an ordered listing of executableinstructions for implementing logical functions. In the example of FIG.2, the instructions in the memory 210 include a suitable operatingsystem (OS) 211. The operating system 211 essentially may control theexecution of other computer programs and provides scheduling,input-output control, file and data management, memory management, andcommunication control and related services.

Additional data, including, for example, instructions for the processor205 or other retrievable information, may be stored in storage 220,which may be a storage device such as a hard disk drive or solid statedrive. The stored instructions in memory 210 or in storage 220 mayinclude those enabling the processor to execute one or more aspects ofthe systems and methods of this disclosure.

The analysis system 120 may further include a display controller 225coupled to a user interface or display 230. In some embodiments, thedisplay 230 may be an LCD screen. In other embodiments, the display 230may include a plurality of LED status lights. In some embodiments, theanalysis system 120 may further include a network interface 260 forcoupling to a network 265. The network 265 may be an IP-based networkfor communication between the analysis system 120 and an externalserver, client and the like via a broadband connection. In anembodiment, the network 265 may be a satellite network. The network 265transmits and receives data between the analysis system 120 and externalsystems. In some embodiments, the network 265 may be a managed IPnetwork administered by a service provider. The network 265 may beimplemented in a wireless fashion, for example, using wireless protocolsand technologies, such as WIFI™, WIMAX™, satellite, or any other. Thenetwork 265 may also be a packet-switched network such as a local areanetwork, wide area network, metropolitan area network, the Internet, orother similar type of network environment. The network 265 may be afixed wireless network, a wireless local area network (LAN), a wirelesswide area network (WAN) a personal area network (PAN), a virtual privatenetwork (VPN), intranet or other suitable network system and may includeequipment for receiving and transmitting signals.

The analysis system 120 accesses the performance data from the cognitivesystem 110 using the network 265. The analysis system 120 furtheranalyzes the performance data to determine the health of the cognitivesystem 110. Additionally, or alternatively, the analysis system 120generates a visualization to depict or represent the health of thecognitive system 110. The analysis system 120 displays the visualization

FIG. 3 illustrates an example visualization of the health of thecognitive system 110 that the analysis system 120 displays using thedisplay 230. The display 230 presents the user-interface to theadministrator 125. The user-interface includes a confidence value scale350 and a time scale 360, among other components to facilitate theadministrator 125 (that is, user of the analysis system 120) toconfigure the visualization. Based on the configuration from theadministrator 125, the analysis system generates and displays thevisualization, which includes a movement pattern, (referred to as flightbehavior) of “boids” to visualize the health of the cognitive system110. A boid is a query-node 305 that the analysis system 120 displays onthe user-interface.

In one or more examples, the analysis system 120 generates thequery-node 305 corresponding to an entry in the query-log of thecognitive system 110. Thus, if the query-log includes multiple entriescorresponding to multiple queries received by the cognitive system 110,the analysis system 120 generates multiple query-nodes corresponding tothe entries in the query-log respectively. The analysis system 120displays the multiple query-nodes on the user-interface to represent thehealth of the cognitive system 120.

The analysis system 120 represents the health by configuring attributesof the query-node 305 according to the data associated with thecorresponding entry in the query-log. The attributes may be animationparameters that cause the query-nodes to be animated. For example, thequery-node includes multiple attributes, such as a position, a color, amovement pattern, a trail 307, and a shape, among others. The analysissystem 120 configures the query-node 305 based on data associated withthe corresponding entry in the query-log, such as confidence value,query-category, and analysis-category, among others. The analysis system120 displays a flock of the multiple query-nodes corresponding to themultiple entries in the query-log respectively (as illustrated in FIG.3). The query-node 305 has a trail 307 that depends on one or moreattribute of the query-node. For example, the color of trail 307 isbased on a type of the question corresponding to the query-node 305.

Thus, each query-node 305 in the flock represents a single question thatthe cognitive system 110 has been asked by the user 105 (or any otheruser). In one or more examples, the analysis system 120 determines amovement pattern of each individual query-node 305 based on theconfidence level of the cognitive system 120 to answer the question thatthe query-node 305 represents. The visualization is further based oncategories of the respective questions. Accordingly, the query-nodesrepresenting questions that the cognitive system 110 is confident inanswering fly in harmony with each other, while the query-nodesrepresenting questions that the cognitive system 110 is not confident inanswering fly more erratically. In one or more examples, the cognitivesystem 110 indicates being confident in answering the question byassigning the question a confidence value above a confidence valuethreshold. Conversely, the cognitive system 110 assigns a confidencevalue below the confidence value threshold to the threshold when thecognitive system is not confident in answering. As a result, when thecognitive system 110 is performing well on most questions, thequery-nodes appear to flock together, suggesting a harmonious system.Alternately, when the cognitive system 110 is performing poorly, amajority of the query-nodes fly chaotically, in different movementpatterns, showing that the cognitive system 110 is in poor health and inneed of improvement. Thus, the analysis system 120 displays the healthof the cognitive system 110 using a flocking behavior of organisms (suchas birds, animals).

In one or more examples, the administrator 125 specifies the confidencethreshold value using the confidence value scale 350. For example, theadministrator 125 slides a marker along a slider to specify theconfidence threshold value. In one or more examples, the confidencevalue scale 350 facilitates the administrator to specify a lowestconfidence value. In response to the selection of the lowest confidencevalue, the analysis system 120 only analyzes the questions with aconfidence value above the lowest confidence value. Accordingly, theconfidence value scale 350 facilitates the administrator to filter thequestions that are used to determine and represent the health of thecognitive system 110. The analysis system 110, thus, does not useoutlier questions that were not formed correctly by the user 105, and inturn preventing false negatives when determining the health of thecognitive system 110.

Additionally, the analysis system 120 facilitates the administrator 125identifying questions of specific types based on the attributes of thequery-nodes. For example, the questions that the cognitive system 110categorizes as questions pertaining to problematic types may bedisplayed using a specific color. For example, as illustrated in FIG. 3,the analysis system 120 displays questions that are related toproblematic topics 310 using a first color and questions that arerelated to unknown domain terms 320 using a second color. It isunderstood that the categorization of the questions may be different andmay include more/fewer/different categories than the illustration.

The visualization provides a multi-pronged interface to facilitate theadministrator 125 to differentiate areas of interest within thequery-log. The visualization provides the administrator 125user-interactivity to zoom into specific subsets and interact/explorewithin the subset, for example to check details of each node in thesubset. Accordingly, the analysis system 120 facilitates theadministrator 125 to filter the data in an interactive manner intosub-categories and zoom into the sub-categories to check the details ofquestions of a particular type or of a particular question. Based on thedetails, the administrator 125 may alter the configuration of thecognitive system 110 in order to improve the performance of thecognitive system 110.

It is understood that although FIG. 3 illustrates the scales in specificpositions using specific user-interface elements, other examples mayinclude the scales at different positions and/or using differentuser-interface elements.

FIG. 4 illustrates a flowchart of an example method for generating thevisualization representing the health of the cognitive system 110 usingthe query-nodes. The analysis system 120 may implement the method in oneor more example. For example, the analysis system 120 receives anidentification of the cognitive system 110, as shown at block 410. Theadministrator 125 provides the identification of the cognitive system110. The identification may be a network identifier, such as an IPaddress, a computer-name, or any other identifier that facilitates theanalysis system 120 to access the cognitive system 120. In one or moreexamples, the administrator 125 may provide access to the query-log ofthe cognitive system 110, instead of the identification of the cognitivesystem 110. For examples, the query-log may be stored at a separatelocation, such as a data repository, from the cognitive system 110.

The analysis system 120 accesses the query-log of the cognitive system110, as shown at block 420. The analysis system 120 generates aquery-node 305 corresponding to each question in the query-log, as shownat block 430. The analysis system 120 may initiate the query-nodes to adefault state. For example, the analysis system 120 may configure theattributes of the query-nodes to predetermined default values. Theattributes of each of the query-nodes, such as the query-node 305include a position, a speed, a course (that is, angle), a color, atrail-color, a shape, an icon, or any other such attributes that affectthe representation of the query-node 305. In one or more examples, thespeed and the course may be integrated into a velocity vector attribute.

The analysis system 120 further computes attribute values for thequery-nodes, as shown at block 440. The analysis system further displaysanimated query-nodes representing health of the cognitive system 110, bydisplaying the query-nodes based on the computed attributes, as shown atblock 460. The analysis system 120 determines the attributes based ondata associated with the corresponding questions of the respectivequery-nodes. The analysis system 120 computes updated attributesperiodically to display an animation representing the health of thecognitive system 110. Alternatively or in addition, the analysis system120 computes the updated attribute values in response to input(s) fromthe administrator 125 viewing and interacting with the animation toidentify performance improvements.

For example, the analysis system 120 determines positions for each ofthe query-nodes, as shown at block 442. In one or more examples, theanalysis system 120 initializes each of the query-nodes with apredetermined coordinate. The query-nodes move from the initial position(coordinate) based on the corresponding attributes that the analysissystem 120 computes in the next frames for the animation.

Alternatively, the analysis system 120 places the query-nodescorresponding to questions of the same category in proximity with eachother. For example, the analysis system 120 computes first positioncoordinates for a first query-node of a first category based on a firstpredetermined position. The first predetermined position may beassociated with the first category. The analysis system 120 may generatea random position, such as a 2D or a 3D coordinate within apredetermined vicinity of the first predetermined position. For example,the predetermined vicinity may include coordinates within apredetermined radius from the first predetermined position. In a similarmanner, the analysis system 120 computes second position coordinates fora second query-node of a second category based on a second predeterminedposition. The analysis system 120 generates the predetermined positionsbased on the number of categories. Accordingly, the analysis system 120initializes the query-nodes at position coordinates based on therespective categories of the questions corresponding to the query-nodes.

The analysis system 120 computes an animation for the query-nodes. Forexample, the analysis system 120 determines a movement vector for eachquery-node, as shown at block 454. The movement vector identifies amovement direction and speed (or animation) of the query-node 305 from acurrent position to a next position, where the query-node is displayedin a next frame. The analysis system 120 determines the movement vectorfor each query-node by computing a separation steer to avoid crowdinglocal flockmates, and alignment steer towards an average heading oflocal flockmates, and cohesion steer to move the query-node toward theaverage position of local flockmates. The three steer values change thespeed and course of the query-node and can be applied in any order toupdate the position of the query-node.

The local flockmates of the query-node 305 are neighboring query-nodesthat are within a predetermined distance from the query-node 305. Theflockmates affect the movement of the query-node 305. The query-node 305and the flockmates are collectively referred to as a flock.

In one or more examples, to compute the cohesion steer, the analysissystem 120 makes the query-node 305 move towards a center of mass of theflock to which the query-node 305 belongs. For example, consider thatthe flock contains N query-nodes, q₁, q₂ . . . q_(n), and that the querynode 305 is the j^(th) query-node (qj) of the flock. In one or moreexamples, the analysis system computes the center of mass (CM) of theentire flock by computing an average position of the N query-nodes. Forexample, CM=(q1.position+q2.position+ . . . +qN.position)/N, whereq_(i).position represents a position vector (for example, 2D coordinate,or 3D coordinate).

In one or more examples, the analysis system 120 computes the center ofmass for computing the movement vector for q_(j)(CM_(j)) of the flock bycomputing an average position of all the other query-nodes of the flock,except the query-node 305. For example,

CM_(j)=(q1.position+q2.position+ . . . +qJ−1.position+bJ+1.position+ . .. +bN.position)/(N−1)

The analysis system 120 then computes the movement of the query-node 305towards the center of mass by displacing the query-node 305 by apredetermined scale. For example, if the predetermined scale moves thequery-node 305 towards the center of mass by 5% in each frame, theanalysis system computes the cohesion steer as a vector offset v₁ forthe query-node 305 by computing v₁=(CM−q_(j).position)*5/100. It isunderstood that CM may be replaced by CMj in other examples. Further,the predetermined scale may be different in other examples, for example1%, 2%, 10%, and so on. In one or more examples, the scale is based onthe confidence value of the question. For example, a question that isassigned a higher confidence value moves faster than a question that isassigned a lower confidence value. Accordingly, query-nodescorresponding to questions with similar confidence values move together.In one or more examples, the scale is based on a difference between theconfidence value and the confidence value threshold. Accordingly, basedon the position and movement vectors, the analysis system 120 separatesthe query-nodes of similar confidence values and depicts the query-nodeswith similar confidence values as moving together.

The analysis system 120 computes the steer to avoid crowding localflockmates to avoid collisions between the query-nodes. Accordingly, ifa neighboring query-node is within a second predetermined distance fromthe query-node 305, the analysis system 120 computes a separation steervalue (S). For example, if the second neighboring query-node is q,S=S−(q.position−q_(j).position). In one or more examples, the analysissystem computes the separation steer value based on each neighboringnode within the second predetermined distance from the query-node 305.For example, Table 1 provides an example pseudo code to compute theseparation steer vector v₂.

TABLE 1 ComputeSeparationVector(query-node qj) { Vector S = 0; D =Second-Predetermined-Distance; FOR EACH neighboring query-node q { IF(|q.position − qJ.position| < D) S = S − (q.position − qj.position) }RETURN S }

The analysis system 120 further computes the alignment steer to makequery-nodes mimic each other's course and speed, and avoid thequery-nodes bouncing around. In one or more examples, the analysissystem 120 the alignment steer vector updates the course and speed ofthe query-node 305. In one or more examples, the analysis system 120computes the alignment steer vector (V) by computing average velocitiesof the neighboring flockmates. V updates both, the course and the speedof the query-node 305. Table 2 provides an example pseudocode forcomputing the alignment steer vector v₃.

TABLE 2 ComputeAlignmentSteer (query-node qJ) { Vector V = 0; FOR EACHneighboring query-node q { A = A + q.velocity − qj.velocity; } V = V /N−1; RETURN V; }

The analysis system 120 updates the speed and course of the query-node305 based on the steer values. For example,qj.velocity=qj.velocity+v₁+v₂+v₃. Using the updated velocity, theanalysis system 120 updates the position of the query-node 305. Forexample, for each unit time the new position of the query-node 305 iscomputed by qj.position=qj.position+qj.velocity.

In one or more examples, additional or different steer vector values arecomputed. In one or more examples, the analysis system 120 scales one ormore of the steer vector values of the query-node 305 based on theattributes of the corresponding question. For example, as describedherein, the v₁ may be scaled based on the confidence value of thequestion. Further, if the corresponding question is from a specificcategory, the steer vectors of the query-node 305 may be scaled in aspecific manner to exhibit a smooth or a chaotic behavior.

In one or more examples, the analysis system 120 uses different sets ofneighboring query-nodes for computing the different steering vectors.For example, the cohesion steer vector v₁ may be computed based on afirst set of neighboring query-nodes, which are within a firstpredetermined distance from the query-node 305; the steer vector v₂ toavoid crowding local flockmates is computed using a second set ofneighboring query-nodes, which are within a second predetermineddistance from the query-node 305; and the alignment steer vector v₃ iscomputed using a third set of neighboring query-nodes, which are withina third predetermined distance from the query-node 305.

Further, the analysis system 120 determines a color for the query-node305 based on the confidence value and the confidence value threshold, asshown at block 444. For example, the analysis system 120 depicts thequery-node 305 using a first color if the corresponding question has aconfidence value below (or equal to) the confidence value threshold, anda second color if the confidence value is above (or equal to) theconfidence value threshold. As described herein, the administrator 125chooses the confidence value threshold using the confidence value scale350. The analysis system 120, thus, depicts the query-nodes that may beconsidered ‘problematic,’ that is having a lower confidence value thanthe confidence value threshold, using the first color. The analysissystem 120 depicts the query-nodes that have confidence value above theconfidence value threshold using the second color.

In one or more examples, the analysis system 120 uses different icons torepresent the query-nodes with different confidence values. For example,query-nodes corresponding to different confidence values may berepresented using respective icons. Alternatively, the icons may beselected based on a difference between the confidence values and theconfidence threshold value.

Thus, based on the color and the flocking of the query-nodes, theanalysis system 120 facilitates the administrator 125 in identifyingwhether the cognitive system 110 is in a ‘healthy’ state or not based onthe confidence value threshold selected by the administrator 125. Forexample, in a healthy state, a majority of the query-nodes hascorresponding confidence values above the selected confidence valuethreshold and is moving in a synchronous pattern. Whereas, in anunhealthy state, the majority of the query-nodes has correspondingconfidence values below the selected confidence value threshold and ismoving chaotically.

The analysis system 120 further facilitates the administrator 125 toidentify questions in specific categories. For example, the analysissystem 120 depicts query-nodes of different categories in a differentmanner, as shown at block 450. For example, the analysis system 120determines a category of the question corresponding to the query-node305, as shown at block 452. The analysis system 120 colors a trail ofthe query-node based on the category of the corresponding question, asshown at block 454. In one or more examples, the analysis system 120scales the steer values of the query-nodes of a first category in afirst manner and the steer values of the query-nodes of a secondcategory in a different manner. Accordingly, query-nodes of the firstcategory move in a first pattern, which is different from a secondpattern in which the query-nodes of the second category move.

For example, as depicted in the example of FIG. 3, in the case of twocategories, the visualization includes two separate flocks correspondingto the two categories (310 and 320) with each query-node in the twocategories moving according to respective confidence values andcategories.

In one or more examples, the analysis system 120 determines the flocksbased on the data in the query-log. For example, the analysis system 120determines categories of the questions from the query-log. In one ormore examples, the cognitive system 110 stores the category of thequestion in the corresponding entry in the query-log. Alternatively, theanalysis system 120 determines the category of the question by parsingthe question from the query-log.

For example, the analysis system 120 flocks the query-nodescorresponding to the questions that the cognitive system 110 was nottrained to answer. For example, the cognitive system may not haveinformation to respond to such questions despite parsing the questions.For example, the set of problematic topics 310 illustrated in FIG. 3represents such a flock of questions that the cognitive system 110parsed successfully and failed to respond to because of lack of trainingon the topic of the questions. The example in FIG. 3 illustrates thatthe cognitive system 110 failed to respond to questions related to thetopic of insurance policies. It is understood that other examples mayidentify a different topic. Further, the cognitive system 110 may failto respond to the questions of more than one topic. Here, a failure torespond indicates that the confidence value assigned to the questions isbelow the confidence value threshold. The administrator 125, uponidentifying such questions may improve the performance of the cognitivesystem 110 by further training the cognitive system 110 in theidentified topic(s). Additionally or alternatively, the cognitive system110 identifies unknown terms in the received questions, as illustratedby the set of unknown domain terms 320 in FIG. 3. The analysis system110 flocks the questions with unknown domain terms. The set of unknowndomain terms 320 represent terms that the cognitive system 110 haddifficulty to parse syntactically or semantically. The administrator125, in response, may update a lexicon of the cognitive system 110.

The analysis system 120 marks the query-nodes corresponding to thequestions in the same category as query-nodes belonging to a flock,using the trails. Thus, the analysis system 120 depicts to theadministrator 125, different issues that the cognitive system 110 isfacing, facilitating the administrator 125 to act accordingly to improvethe performance of the cognitive system 110.

The analysis system 120 displays the animated query-nodes representinghealth of the cognitive system 110 according to the attributes computed,as shown at block 470. In one or more examples, the analysis system 120updates the attributes of the query-nodes and displays the query-nodesaccording to a predetermined frequency. The predetermined frequencyfacilitates an animated visualization representing the health of thecognitive system 110.

The analysis system 120 further facilitates the administrator 125 tointeract with the visualization that includes the animated query-nodes.For example, as illustrated in FIG. 5, if the administrator selects thequery-node 305, the analysis system 120 displays connectors 510 betweenthe query-nodes of the category to which the query-node 305 belongs. Forexample, in the example of FIG. 5, the administrator selects thequery-node 305 that belongs to the problematic topics category 310.Accordingly, the analysis system 120 depicts the connectors between thequery-nodes of the category 310. Alternatively or in addition, theanalysis system 120 lists the questions corresponding to the query-nodesof the selected category. If the administrator 125 selects one of thequestions listed, the analysis system 120 displays a parsed view 605 ofthe question, as shown at FIG. 6. The parse view displays a visual breakdown of how the cognitive system 110 parsed the question.

Further, the analysis system 120 facilitates the administrator to zoomin and explore a specific category of questions by exploring theconnected query-nodes. For example, the administrator may select tozoom-in or zoom-out of the connected query-nodes to view details of thecorresponding questions. Alternatively or in addition, the administratormay select to switch the connectors off by selecting another query-nodeor an open space. In one or more examples, the administrator 125 inputsselection via an input-device such as a mouse, a keyboard, a joystick, atouchscreen, or any other input device.

In one or more examples, the administrator 125 selects a subset of thequestions from the query-log to be represented in the visualization. Forexample, the administrator 125 specifies a time range using the timescale 360. For example, the administrator 125 may select a starting dateof Aug. 6, 2015. Based on the selection, the analysis system 110 onlyuses the questions that the cognitive system 110 received after Aug. 6,2015 (see FIG. 3). Alternatively or in addition, the administrator 125may select an ending date for the time range, for example Dec. 31, 2015.Accordingly, the analysis system 120 only uses questions that werereceived between Aug. 6, 2015, and Dec. 31, 2015. It is understood thatthe administrator 125 may select any other starting date and/or endingdate for the time range.

Accordingly, the analysis system 120 represents the health of thecognitive system 110 based on a selected subset of the query-log. Theanalysis system 120 generates and displays a visualization that includesone or more flocks of query-nodes. Each query-node in a flock representsa single question that the cognitive system 110 has been asked by theuser 105. The movement behavior of each individual query-node is basedon the confidence assigned by the cognitive system corresponding to anability to answer the question. The query-nodes representing questionsthat the cognitive system 110 is confident in answering fly in harmony,while query-nodes representing questions that the cognitive system 110is not confident in answering fly more erratically. Thus, when thecognitive system 110 is performing well on at least a predeterminedproportion of questions, the query-nodes appear to flock together,suggesting a harmonious cognitive system 110. Alternately, when thecognitive system 110 is performing poorly, at least a predeterminedproportion of the query-nodes fly chaotically, showing that thecognitive system 110 is in poor health and in need of improvement.

Thus, the technical solutions described herein determine and depicthealth of a cognitive system that answers user submitted questions usingartificial intelligence. In one or more examples, the health is depictedusing computer animation, such as computer animated flocking behavior torepresent the health of the cognitive system. Further, the visualrepresentation of the health is interactive and facilitates anadministrator of the cognitive system to interact with the performancedata that used to generate the visual representation. For example, theinteraction facilitates the administrator to identify specific problemshampering the performance of the cognitive system and improve theperformance accordingly. For example, the administrator may impartadditional/different training to the cognitive system, may expand alexicon of the cognitive system, and/or may undertake any otherimprovement. Accordingly, the technical solutions described hereinprovide a free flowing computer generated animation that depicts thehealth of the cognitive system that uses dynamic clustering tofacilitate the administrator to identify performance issues with thecognitive system and take corrective actions. The technical solutionsfurther facilitate the administrator to determine details of specificperformance issues by interacting with the computer generated animation.

The present technical solutions may be a system, a method, and/or acomputer program product at any possible technical detail level ofintegration. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent technical solutions.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present technical solutions may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present technicalsolutions.

Aspects of the present technical solutions are described herein withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems), and computer program products according toembodiments of the technical solutions. It will be understood that eachblock of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present technical solutions. In this regard, eachblock in the flowchart or block diagrams may represent a module,segment, or portion of instructions, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). In some alternative implementations, the functions noted inthe blocks may occur out of the order noted in the Figures. For example,two blocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts or carry outcombinations of special purpose hardware and computer instructions.

A second action may be said to be “in response to” a first actionindependent of whether the second action results directly or indirectlyfrom the first action. The second action may occur at a substantiallylater time than the first action and still be in response to the firstaction. Similarly, the second action may be said to be in response tothe first action even if intervening actions take place between thefirst action and the second action, and even if one or more of theintervening actions directly cause the second action to be performed.For example, a second action may be in response to a first action if thefirst action sets a flag and a third action later initiates the secondaction whenever the flag is set.

To clarify the use of and to hereby provide notice to the public, thephrases “at least one of <A>, <B>, . . . and <N>” or “at least one of<A>, <B>, <N>, or combinations thereof” or “<A>, <B>, . . . and/or <N>”are to be construed in the broadest sense, superseding any other implieddefinitions hereinbefore or hereinafter unless expressly asserted to thecontrary, to mean one or more elements selected from the groupcomprising A, B, . . . and N. In other words, the phrases mean anycombination of one or more of the elements A, B, . . . or N includingany one element alone or the one element in combination with one or moreof the other elements which may also include, in combination, additionalelements not listed.

It will also be appreciated that any module, unit, component, server,computer, terminal or device exemplified herein that executesinstructions may include or otherwise have access to computer readablemedia such as storage media, computer storage media, or data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Computer storage media may includevolatile and non-volatile, removable and non-removable media implementedin any method or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Such computer storage media may be part of the device or accessible orconnectable thereto. Any application or module herein described may beimplemented using computer readable/executable instructions that may bestored or otherwise held by such computer readable media.

The descriptions of the various embodiments of the present technicalsolutions have been presented for purposes of illustration, but are notintended to be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application, or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer implemented method for representinghealth of a cognitive system, the method comprising: displaying ananimated set of icons, each icon representing a question input to thecognitive system, wherein each icon has a respective movement pattern;receiving a selection of a first icon from the animated set of icons;and in response, identifying a category of a first questioncorresponding to the first icon; determining a subset of the iconscorresponding to question from the category; and displaying connectorsbetween the subset of the icons.
 2. The computer implemented method ofclaim 1, wherein the animated set of icons moves in a synchronouspattern in response to representing a healthy cognitive system.
 3. Thecomputer implemented method of claim 1, wherein a speed of movement of arespective icon is related to a confidence level of a question.
 4. Thecomputer implemented method of claim 1, wherein an icon representing aquestion of a category is presented proximate to icons representingother questions of the category.
 5. The computer implemented method ofclaim 1, wherein a color of a respective icon is related to a confidencelevel of a question.
 6. The computer implemented method of claim 1,wherein icons representing questions of a first category of questionsare presented having a first trail type.
 7. The computer implementedmethod of claim 1, further comprising: in response to receiving theselection of the first icon from the set, displaying a parse view of thefirst question.
 8. The computer implemented method of claim 1, furthercomprising: in response to receiving the selection of the first iconfrom the set, displaying a list of questions corresponding to the subsetof the icons.
 9. A system for representing health of a cognitive system,the system comprising: a memory; a user-interface; and a processorcoupled with the memory and the user-interface, the processor configuredto: display, via the user-interface, an animated set of icons, each iconrepresenting a question input to the cognitive system, wherein each iconhas a respective movement pattern; receive, via the user-interface, aselection of a first icon from the set; and in response, identify acategory of a first question corresponding to the first icon; determinea subset of the icons corresponding to question from the category; anddisplay, via the user-interface, connectors between the subset of theicons.
 10. The system of claim 9, wherein an icon representing aquestion of a category is presented proximate to icons representingother questions of the category.
 11. The system of claim 9, whereinicons representing questions of a first category of questions arepresented having a first trail type.
 12. The system of claim 9, whereinthe processor is further configured to: in response to receiving theselection of the first icon from the set, cause the user-interface todisplay a parse view of the first question.
 13. The system of claim 9,wherein the processor is further configured to: in response to receivingthe selection of the first icon from the set, cause the user-interfaceto display a list of questions corresponding to the subset of the icons.14. A computer program product for visually depicting health of acognitive system, the computer program product comprising a computerreadable storage medium, the computer readable storage medium comprisingcomputer executable instructions, wherein the computer readable storagemedium comprises instructions to: display, via a user-interface, ananimated set of icons, each icon representing a question input to thecognitive system, wherein each icon has a respective movement pattern;receive, via the user-interface, a selection of a first icon from theset; and in response, identify a category of a first questioncorresponding to the first icon; determine a subset of the iconscorresponding to question from the category; and display, via theuser-interface, connectors between the subset of the icons.
 15. Thecomputer program product of claim 14, wherein an icon representing aquestion of a category is presented proximate to icons representingother questions of the category.
 16. The computer program product ofclaim 14, wherein icons representing questions of a first category ofquestions are presented having a first trail type.
 17. The computerprogram product of claim 14, wherein computer readable storage mediumfurther comprises instructions to: in response to receiving theselection of the first icon from the set, cause the user-interface todisplay a parse view of the first question.
 18. The computer programproduct of claim 14, wherein computer readable storage medium furthercomprises instructions to: in response to receiving the selection of thefirst icon from the set, cause the user-interface to display a list ofquestions corresponding to the subset of the icons.
 19. The computerprogram product of claim 14, wherein a speed of movement of a respectiveicon is related to a confidence level of a question.
 20. The computerprogram product of claim 14, wherein the computer readable storagemedium further comprises instructions to zoom-in and zoom-out from thesubset of the icons corresponding to the first category.